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FOREWORD 


The  Educational  Technology  and  Training  Simulation  Technical  Area  of 
the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
conducts  research  to  support  training  concepts,  instructional  system  de- 
velopment, and  technology  assessment. 

One  system  being  considered  to  facilitate  dissemination  of  research 
results  in  these  areas  is  the  Automated  Data  on  Instructional  Technology 
(ADIT)  System.  ADIT  development  was  initiated  by  the  Air  Force  Human  Re- 
sources Laboratory  (AFHRL)  and  has  been  jointly  funded  by  AFHRL,  ARI,  the 
Navy  Personnel  Research  and  Development  Center  (NPRDC) , and  the  Defense 
Advanced  Research  Projects  Agency  (DARPA)  since  FY  75.  ADIT  is  subsumed 
under  Air  Force  Office  of  Scientific  Research  Project  2313,  "Human  Resour- 
ces Task  3,  Human  Factors  in  System  Design."  ARI ' s portion  of  the  effort 
is  funded  as  part  of  ARI  project  2Q761101B74F , "Personnel  Performance  and 
Training."  

Although  the  full  system  will  not  be  completed  until  FY  79,  a pre- 
liminary evaluation  was  requested  by  the  Office  of  the  Director  of  Defense 
Research  and  Engineering. 

Special  thanks  go  to  Mr.  John  Ferguson,  ARI  Librarian,  for  his  con- 
siderable assistance  during  the  conduct  of  this  evaluation. 
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ARMY  RESEARCH  INSTITUTE  EVALUATION  OF  AUTOMATED  DATA  ON  INSTRUCTIONAL 
TECHNOLOGY  (ADIT) 


BRIEF 


Requirement: 

To  conduct  a "limited"  test  for  assessing  the  comparative  usefulness 
of  the  records  in  Automated  Data  on  Instructional  Technology  (ADIT) . An 
integral  part  of  this  effort  was  to  evaluate: 

1.  The  stand-alone  quality  of  the  ADIT  Guide  to  Use  and 

2.  The  ease  of  on-line  data  retrieval. 

The  objective  was  to  provide  an  assessment  of  the  User's  Guide  and  opera- 
tions for  retrieving  information  from  the  data  base,  as  well  as  recommen- 
dations for  improvement  of  both  if  necessary. 

Procedures : 

Seven  research  psychologists,  grades  GS-11  and  14,  participated  in 
the  study.  Each  performed  a series  of  tasks  that  included: 

1.  Reading  the  ADIT  Guide  to  Use, 

2.  Performing  search  strategy  exercises, 

3.  Performing  the  ADIT  Operations  Test,  and 

4.  Completing  two  questionnaires:  (a)  Evaluation  of  ADIT  Guide 
to  Use  and  (d)  Evaluation  of  ADIT. 

One  briefing  session  was  held,  during  which  participants  were  told 
the  purpose  of  the  evaluation  and  taught  how  to  operate  the  two  terminals 
used  in  the  study.  All  participants  kept  logs  to  record  the  time  spent 
both  in  off-line  reading  and  preparation  of  search  strategies,  as  well  as 
in  actual  ^ata  retrieval.  Participants  were  encouraged  to  provide  written 
suggestions  for  improving  the  data  base  itself,  methods  for  accessing  in- 
formation, and  the  User  Guide  Format. 

The  ADIT  operations  test  was  a 10-part,  sel f-administered  on-line 
test  that  sought  to  determine  the  efficiency  of  the  student's  search 
strategies  and  the  appropriateness  of  the  choices  of  segments  to  be 
displayed.  Scoring  was  based  upon  guidelines  provided  by  AFHRL.  The 


questionnaires  that  evaluated  the  guide  and  evaluated  the  ADIT  system 
used  a 10-interval  scale  with  a centered  zero  reference.  Measurements 
were  taken  to  the  nearest  whole  number. 


Findings : 

1.  The  attitudes  toward  the  ADIT  system  per  se  were  generally 
favorable.  Participants  felt  that  the  system  would  be  par- 
ticularly useful  for  educational  researchers,  administrators, 
planners,  and  developers. 

2.  The  information  in  ADIT  files  was  found  to  be  comprehensive 
and  useful.  Participants  believed  that  data  could  be  re- 
trieved from  ADIT  more  quickly  than  from  either  original 
sources  or  alternative  data  bases,  such  as  Defense  Documen- 
tation Center  (DDC) . 

3.  Attitudes  toward  the  ADIT  Guide  to  Use  were  less  favorable. 
Participants  indicated  that  (a)  the  Guide  did  not  contain  suf- 
ficient information,  (b)  the  instructions  were  not  written 
with  sufficient  clarity,  and  (c)  the  questions  in  the  exer- 
cises were  of  limited  usefulness. 


Utilization  of  Findings: 

Responses  to  the  questionnaires,  as  well  as  protocol  analysis  of 
participants’  operations  tests,  revealed  that  most  of  the  problems  en- 
countered and  most  of  the  errors  on  the  test  were  attributable  to  short- 
comings in  the  User's  Guide.  It  was  generally  believed  that  a more 
meaningful  evaluation  of  the  system  itself  could  be  conducted  if  the 
guide  were  revised  to  incorporate  some  of  the  suggested  modifications. 

It  is  anticipated  that  the  recommendations  for  improvement  of  the  User's 
Guide  that  emerged  from  this  effort  will  be  incorporated  with  those  from 
concurrent  AFHRL  and  NPRDC  evaluations  to  produce  a revised  User's  Guide. 

One  of  the  specific  recommendations  made  in  this  report  has  already 
been  accomplished:  namely,  the  system  now  permits  the  use  of  arithmetic 
operators  in  segment  28. 
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ARMY  RESEARCH  INSTITUTE  EVALUATION  OF  AUTOMATED  DATA 
ON  INSTRUCTIONAL  TECHNOLOGY  (ADIT) 


INTRODUCTION 

Automated  Data  on  Instructional  Technology  (ADIT) , which  began  in 
May  1974,  is  a computerized  file  of  technical  (research)  information  on 
instruction  and  learning  for  use  by  anyone  within  the  Department  of  De- 
fense involved  with  Training  and  instructional  System  Development  (ISD) . 
The  Air  Force  Human  Resources  Laboratory  (AFHRL)  is  accomplishing  the 
effort  with  support  from  the  Air  Force  Office  of  Scientific  Research. 
Additional  participative  support  has  been  provided  by  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI ) , the  Navy  Person- 
nel Research  and  Development  Center  (NPRDC) , and  the  Defense  Advanced 
Research  Projects  Agency  (DARPA) . 

The  file  consists  of  exceptionally  detailed  evaluative  abstracts 
of  technical  reports,  journal  articles,  books,  etc.,  relating  to  training 
and  ISD.  Each  entry  contains  up  to  35  pieces  of  information  in  addition 
to  the  abstract  and  normal  bibliographic  citation.  These  abstracts  are 
intended  to  be  so  comprehensive  that  original  sources  will  be  unnecessary 
for  most  purposes. 

Access  to  the  file  is  via  an  on-line,  interactive,  full-text  re- 
trieval system  (INFOCEN)  located  at  Wright-Patterson  Air  Force  Base. 

Any  dial-up  terminal  of  common  configuration  may  be  used,  with  the  only 
added  expense  being  telephone  toll  charges.  (Later  plans  call  for  mak- 
ing ADIT  available  less  expensively  through  commercial  nets.) 

Nearly  11,000  abstracts  are  on-line,  with  500  more  scheduled  for  in- 
clusion by  1 October  1978.  It  is  estimated  that  the  total  system  will 
consist  of  14,000  entries.  After  the  initial  phase,  a capability  will 
be  added  for  using  the  data  base  to  obtain  prescriptive  guidance.  The 
last  phase  will  include  refinements  that  will  permit  users  to  query  the 
system  about  specific  instructional  design  issues,  using  constrained 
natural  language  interaction.  Once  the  system  becomes  functional,  ad- 
ditional relevant  information  (not  yet  formally  reported)  will  be  in- 
cluded as  part  of  the  updating  and  file  maintenance  process. 

BACKGROUND 

On  10  June  1976  the  office  of  the  Director  of  Defense  Research  and 
Engineering  requested  an  evaluation  of  the  potential  usefulness  of  the 
ADIT  system.  An  evaluation  team,  consisting  of  one  member  from  each 
participating  organization  (AFHRL,  ARI,  and  NPRDC)  was  convened  to  draw 
up  an  evaluation  plan.  Given  the  limited  time,  personnel,  and  funding 
resources,  it  was  agreed  that  a small-scale  test  be  conducted  to  examine 
the  utility  of  ADIT  files,  the  ease  of  on-line  retrieval,  and  the  value 
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of  the  ADIT  Guide  to  Use  as  a teaching  instrument.  The  guide,  opera- 
tions test,  and  two  questionnaires  used  in  the  evaluation  were  developed 
by  AFHRL.  An  effort  would  be  made  to  obtain,  whenever  possible,  sub- 
jects who  would  represent  different  types  of  users.  In  the  case  of  ARI , 
no  persons  experienced  in  school  instruction  (and  onlv  one  experienced 
in  instructional  technology)  were  available.  None  had  ever  had  any  di- 
rect responsibility  for  planning,  developing,  or  managing  military  in- 
struction, but  all  had  been  involved  in  training-related  research. 
Essentially,  the  sample  was  composed  of  individuals  with  advanced  de- 
grees in  psychology. 


DESCRIPTION  OF  ADIT  GUIDE  TO  USE 

The  ADIT  Guide  was  prepared  as  a stand-alone,  self-instructional 
text.  Several  instructional  features  have  been  included  as  appendixes. 

The  guide  contains  a section  that  describes  the  system  and  others  that 
give  general  procedures  for  starting  up,  signing  in,  searching,  retriev- 
ing, processing,  and  terminating.  More  complex  operating  procedures  are 
handled  in  later  sections  of  the  text.  These  include  several  strategies, 
e.g.,  formulating  the  request  and  organizing  the  output,  browsing,  modi- 
fying requests,  etc.  The  appendixes  contain  a complete  listing  of  all 
ADIT  file  segment  numbers  (with  explanatory  notes)  , a directory  of  gen- 
eral terms,  review  questions  on  system  description  and  procedures,  a 
checklist  for  initial  practice  operation,  and  10  practice  search  prob- 
lems with  suggested  solutions. 

ADIT  EVALUATION 

Procedure 

Each  participant  was  given  a copy  of  the  User’s  Guide  and  a set  of 
forms  to  complete,  including  a log  for  time  spent  both  on-line  and  off- 
line. All  subjects  were  required  to  perform  the  same  series  of  tasks. 

The  sequence  was  specified,  but  moderate  departures,  because  of  schedul- 
ing problems,  etc.,  were  anticipated,  and  allowances  were  made  for  them. 

The  scheduled  activities  are  listed  below  in  the  suggested  order  of 
performance: 

1.  Complete  an  "ADIT  Evaluation  Participation  Record." 

2.  Read  the  "ADIT  Guide  to  Use"  to  page  28. 

3.  Perform  the  "Checklist  for  Initial  Practice  Operation"  (Appendix 
D of  the  guide) . 

4.  Complete  reading  the  "ADIT  Guide  to  Use."  | 

5.  Perform  the  search  strategy  exercise  in  Appendix  E of  the  Guide 


and  verify  solutions  at  the  terminal. 


6.  Perform  the  "ADIT  Operation  Test." 

7.  Complete  the  questionnaire,  "Evaluation  of  the  "ADIT  Guide  to 
Use.  " 

8.  Work  up  to  30  minutes  on  self-identified  problem (s)  of  per- 
sonal interest. 

9.  Complete  the  questionnaire,  "Evaluation  of  ADIT." 

Seven  scientists  (research  psychologists,  GS-11  and  14)  participated 
in  the  evaluation.  None  had  any  previous  experience  with  on-line  access- 
ing of  data  bases,  but  they  had  at  least  limited  experience  with  computer 
systems.  One  briefing  session  was  held,  during  which  the  participants 
were  informed  about  the  nature  of  evaluation,  and  materials  were  distrib- 
uted (see  appendixes) . An  experienced  user  of  the  ADIT  system  was  onsite 
at  all  times  during  the  evaluation  period.  The  only  assistance  offered, 
however,  was  to  refer  individuals  seeking  guidance  to  the  appropriate 
pages  in  the  Guide,  in  addition  to  providing  procedural  information  with 
regard  to  operation  of  the  terminal. 

Ideally,  each  participant  was  to  set  aside  two  consecutive  working 
days  to  perform  the  bulk  of  the  activities.  In  actual  practice,  this  did 
not  prove  possible;  therefore,  the  period  of  performance  ranged  from  10 
to  15  days. 

Experienced  users  from  AFHRL,  Wright- Patterson  Air  Force  Base,  ad- 
vised that  actual  time  spent  at  the  terminal  during  any  given  session 
should  be  approximately  30  minutes.  Because  it  frequently  took  that  long 
just  to  establish  contact  with  Data  Central,  it  was  not  unusual  for  par- 
ticipants to  spend  2 or  more  hours  during  each  sitting  at  the  terminal. 
There  is  no  way  to  determine  accurately  the  extent  to  which  this  may  have 
affected  performance  in  terms  of  accuracy  and  efficiency.  Expediency 
dictated,  however,  that  this  was  the  only  way  to  accomplish  the  work 
within  a reasonable  time  frame. 

The  evaluation  plan  called  for  each  user  to  work  independently  at 
all  times.  Because  of  the  press  of  other  activities  and  the  availability 
of  only  two  terminals,  it  was  occasionally  necessary  for  two  or  more  re- 
searchers to  work  together,  especially  during  the  practice  exercises.  To 
insure  that  researchers  had  worked  out  the  search  strategies  on  their  own, 
a joint  input  was  made  at  the  terminal  only  after  it  was  determined  that 
these  search  plans  were  essentially  identical.  In  such  cases,  the  time 
spent  was  recorded  for  each  participant.  Every  attempt  was  made  to  re- 
duce or  eliminate  biasing  of  the  experimental  results. 


Findings 


Operations  Test.  The  operations  test  was  a job-performance  test 
intended  for  evaluating  proficiency  in  the  use  of  ADIT.  Thirteen  tasks 
were  included  (see  Appendix  A) . Instructions  called  for  participants 
to  formulate  a search  strategy,  select  segments  to  display,  and,  where 
called  for,  perform  the  search  and  record  the  answers.  Different  ques- 
tions had  different  point  values,  depending  upon  the  complexity  of  the 
task.  Maximum  possible  score  was  60.  Part  of  the  test  required  parti- 
cipants to  plan  and  execute  a search  strategy  on  a topic  of  their  own 
choosing,  a task  that  most  found  quite  difficult.  Table  1 shows  the 
scores  for  each  question,  the  total  score  for  each  participant,  and  the 
total  time  each  spent  performing  all  the  formal  requirements  for  the 
study. 

A comparison  of  scores  on  the  operations  test  with  total  amount  of 
time  spent  indicates  that,  for  the  most  part,  higher  scores  were  associ- 
ated with  more  time  spent  in  study  and  practice.  Final  scores  ranged 
from  37  to  55,  with  a mean  of  45.9.  All  particpants  lost  at  least  some 
points  as  a result  of  not  following  instructions  completely  in  all 
instances . 

Most  participants  appeared  to  find  guestions  1,  2,  3,  5,  and  8 rela- 
tively easy.  As  the  questions  became  more  complex,  though,  the  number 
and  variety  of  errors  increased. 

Careful  examination  of  the  participant-response  protocols  often 
yielded  clues  as  to  probable  causes  of  confusion.  Errors  tended  to  be 
consistent  across  subjects  and,  in  many  instances,  could  be  at  least 
partially  traced  to  ambiguities  or  omissions  in  the  guide. 

Question  6 proved  difficult,  possibly  because  most  participants 
failed  to  enter  appropriate  alternative  search  terms. 

The  greatest  variability  was  manifested  in  questions  9a-9d.  The 
total  number  of  answers  provided  by  the  system  was  largely  a function 
of  whether  participants  entered  their  search  statements  via  modifica- 
tions. A particular  problem  area  was  question  9d,  which  the  scoring 
key  treated  as  a modification  when  actually  it  might  have  been  more  pro- 
ductive to  enter  a completely  new  search  statement. 

Performance  was  notably  poor  on  question  10,  which  required  par- 
ticipants to  design  and  carry  out  a detailed  search  plan  on  individual- 
ized training.  Examination  of  responses  seemed  to  indicate  that 
participants  had  not  mastered  the  skills  needed  to  perform  a thorough 
search.  During  postevaluation  debriefing  sessions,  however,  it  became 
evident  that  at  least  part  of  the  poor  performance  could  be  ascribed 
to  lack  of  enthusiasm  for  the  task  rather  than  lack  of  ability. 
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ADIT  Operations  Test  Scores 
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Evaluation  of  ADIT.  Participants  each  completed  a 12-part  ques- 
tionnaire that  evaluated  the  ADIT  system  and  operating  procedures. 

Table  2 provides  summary  scores  and  means  for  all  questions  that  used 
numerical  ratings.  Scoring  was  along  a 10-interval  scale  with  a cen- 
tered zero  reference,  measured  to  the  nearest  whole  number.  A complete 
listing  of  all  questions  and  responses  can  be  found  in  Appendix  B. 

Procedures  for  using  ADIT  were  rated  as  relatively  simple  (ques- 
tion 1,  Appendix  B) . Estimates  regarding  the  amount  of  time  needed  for 
practice  at  the  terminal  (in  order  to  become  proficient  in  the  use  of 
ADIT)  ranged  from  a low  of  1 hour  to  a high  of  40  hours  (question  2, 
Appendix  B) . The  system  records  obtained  during  the  test  were  rated  as 
more  than  adequate  for  use  in  reviewing  research  on  the  topics  involved 
(question  3,  Appendix  B) . Additional  comments  by  participants,  however, 
indicated  a strong  personal  preference  for  referring  to  original  docu- 
ments, particularly  for  the  most  directly  relevant  materials.  This  is 
less  a criticism  of  the  system  than  a predictable  bias  of  professional 
researchers  in  favor  of  original  sources.  Participants  thought  that 
the  records  contained  information  that  would  be  equally  useful  to  plan- 
ners of  service  training  courses  and  to  those  reviewing  research  (ques- 
tion 4,  Appendix  B) . 

All  participants  indicated  that  if  they  had  ready  access  to  a ter- 
minal, they  would  use  its  services.  Frequency  estimates  ranged  from 
one  to  eight  times  a month,  and  time  estimates  varied  from  5 minutes  to 
3 hours  (question  5,  Appendix  B) . 

On  topics  relating  to  instructional  design,  ADIT  was  seen  as  offer- 
ing a very  appreciable  savings  in  time  over  both  conventional  library 
searches  and  the  DDC  (question  6,  Appendix  B) . 

Participants  felt  that  when  the  document  coverage  is  completed,  the 
records  would  be  somewhat  useful  to  personnel  involved  in  administering, 
planning,  and  developing  instructional  systems  and  devices,  but  more  use- 
ful to  DOD  researchers  in  instructional  technology  (questions  7 and  8, 
Appendix  B) . Modifying  the  system  to  permit  querying  ADIT  for  principles 
relating  to  instructional  design  and  for  specific  recommendations  on  par- 
ticular problems  in  that  area  was  thought  to  be  of  more  value  to  research- 
ers than  to  administrators,  planners,  or  developers  (questions  9a  and  9b, 
Appendix  B) . Asked  to  indicate  support  for  ADIT  as  a taxpayer  rather 
than  a scientist,  participants  tended  to  favor  rather  than  object  to  the 
project  (question  10,  Appendix  B) . 

Evaluation  of  ADIT  Guide  to  Use.  Each  participant  completed  an 
eight-part  questionnaire  that  evaluated  the  ADIT  Guide  to  Use.  Table  3 
provides  summary  scores  and  means  for  all  questions  that  used  numerical 
ratings.  Scoring  was  also  along  a 10-interval  scale.  A complete  list- 
ing of  all  questions  and  responses  can  be  found  in  Appendix  C.  Pages 
1-28  contain  the  fundamental  operating  procedures  for  ADIT.  Participants 
spent  a mean  of  76  minutes  reading  this  portion  of  the  guide  (question 
la.  Appendix  C) . 
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Table  2 


Scores  and  Means  for  Evaluation  of  ADIT  Questionnaire 


Table  3 

Score  and  Means  for  Evaluation  of  ADIT  Guide  to  Use 


Appendix  D of  the  guide  (Checklist  for  initial  Practice  Operation) 
took  a mean  of  108  minutes  to  complete,  with  a range  of  42  to  175  min- 
utes (question  lb.  Appendix  C) . 

Although  the  findings  show  that  participants  spent  an  average  of 
115  minutes  to  complete  reading  the  guide  (question  lc.  Appendix  C) , 
this  could  be  an  underestimate  because  it  may  not  reflect  all  of  the 
time  spent  looking  up  information  following  the  initial  reading.  Par- 
ticipants stated  that  because  this  time  was  spent  at  irregular  intervals, 
the  logs  may  not  document  accurately  all  time  spent. 

Appendix  E of  the  guide  contained  10  practice  search  problems.  Al- 
though the  guide  offered  assistance  in  the  form  of  suggested  solutions, 
the  amount  of  time  spent  suggests  that  participants  opted  to  try  their 
own  solutions  (question  Id,  Appendix  C) . The  task  that  took  longest  to 
perform  was  the  operations  test;  mean  time  was  382  minutes  (question  le , 
Appendix  C) . All  participants  reported  a need  to  refer  to  the  guide 
frequently  during  the  course  of  the  test.  In  many  instances,  several 
attempts  were  made  on  questions  before  an  answer  was  deemed  satisfactory. 

A wide  range  of  topics  was  selected  by  participants  as  areas  to 
work  on  purely  for  self-interest.  Times  for  this  activity  ranged  from 
5 minutes  to  more  than  1-1/2  hours  (question  If,  Appendix  C) . 

The  clarity  of  the  guide  (for  purposes  of  learning)  was  rated  rela- 
tive to  similar  materials.  Ratings  were  closer  to  the  "very  confusing" 
end  of  the  scale  than  they  were  to  the  "very  clear"  portion  (question  2, 
Appendix  C) . The  length  of  the  guide  was  seen  as  somewhat  "too  short" 
(question  3,  Appendix  C) , and  the  information  provided  was  considered 
"too  little"  (question  4,  Appendix  C) . 

The  guide  was  also  seen  as  being  deficient  in  terms  of  providing 
the  i.- formation  necessary  to  use  ADIT  at  the  terminal  (question  8,  Ap- 
pendix c:  In  spite  of  these  shortcomings,  however,  it  was  felt  that 

an  individual  who  had  mastered  the  guide  would  be  reasonably  competent 
to  operate  the  ADIT  system  (question  5,  Appendix  C) . Estimates  for  how 
much  time  should  be  spent  studying  the  guide  by  an  individual  with  no 
computer  experience  but  with  some  knowledge  of  training  ranged  from  2 
to  3 hours  to  as  much  as  8 hours  (question  6,  Appendix  C) . 

Questions  in  Appendix  C of  the  guide  were  not  viewed  as  being  par- 
ticularly valuable  in  learning  the  materials  (question  7,  Appendix  C) . 

Problems  Encountered . Perhaps  the  most  frustrating  aspect  of  the 
ADIT  evaluation  was  the  long  delays  experienced  in  gaining  access  to 
the  data  base.  Certain  problems  were  apparently  due  to  faulty  communi- 
cation lines,  particularly  the  Autovon  lines.  Waiting  periods  of  15  to 
30  minutes  or  more  were  not  uncommon,  especially  in  the  mornings  and 
late  afternoons.  Consequently,  the  tasks  took  much  longer  to  complete 
than  originally  anticipated.  During  a considerable  portion  of  the  evalu- 
ation period,  there  were  many  system  bugs.  For  example,  answers  would 

8 


i 


be  rejected  upon  initial  entry,  only  to  be  accepted  upon  reentry.  The 
system  also  would  refuse  occasionally  to  accept  the  10-digit  code  num- 
ber, and  at  times  it  would  accept  a typed-out  name  for  a segment  but  not 
the  corresponding  code  number. 

There  were  times  when  the  system  responded  "missing  operator"  when, 
in  fact,  none  was  missing.  At  other  times,  the  system  would,  in  the  mid- 
dle of  an  operation,  return  to  file  without  having  been  instructed  to  do 
so.  With  great  regularity,  complex  search  statements  would  cause  the 
system  to  respond,  "your  request  is  being  processed"  repeatedly,  until 
the  user  would  hand  up  in  frustration  and  start  over  again.  Sometimes, 
a complex  request  would  yield  "no  entries"  or  "too  much  information  given" 
when  the  very  same  request  entered  by  way  of  modification  would  produce 
answers.  The  preceding  discussion  of  problems  clearly  suggests  a need 
for  further  debugging  and  a mechanism  for  monitoring  the  system  to  mini- 
mize such  problems  in  the  future. 

The  following  summarizes  the  most  frequently  occurring  comments  re- 
garding deficiencies  in  the  User's  Guide: 

The  use  of  quotation  marks  is  confusing.  The  guide  frequently  puts 
quotation  marks  around  words  to  be  entered  as  responses,  although  the 
quotes  are  not  to  be  entered.  It  would  be  helpful  if  this  were  stated 
clearly  in  the  guide. 

Obeying  the  instructions  (p.  32)  for  entering  the  name  of  an  author 
(using  the  initials  for  greater  precision)  in  some  instances  yielded  few- 
er responses  than  just  the  author's  last  name  alone.  In  the  worst  case, 
entering  an  author's  name  and  initials  would  produce  "no  response"  when 
there  were,  indeed,  one  or  more  answers  when  the  last  name  was  entered 
alone.  This  proved  annoying  to  users  who  assumed  that  the  guide's  di- 
rections were  the  best  ones  to  use. 

The  guide  did  not  indicate  that  the  six  arithmetic  operators  do  not 
work  for  file  segment  28  (date) . Examination  of  participant  protocols 
reveals  that  everyone  made  the  same  erroneous  assumption  regaiding  seg- 
ment 28.  The  guide  is  also  not  clear  on  how  to  proceed  when  the  system 
responds  as  if  a perfectly  legitimate  segment  name  were  invalid,  nor 
does  it  indicate  which  statements  are  acceptable  after  the  system  re- 
sponds, "will  attempt  to  continue."  This  resulted  in  a frequently  fruit- 
less trial-and-error  process. 

At  one  point  in  the  guide,  users  were  told  that  a search  request 
could  be  modified  by  entering,  "MODIFY"  (see  p.  22).  There  is,  in  addi- 
tion, a recursive  command,  "$$MODIFY,"  which  returns  the  system  to  the 
message  "ADD  NUMBER  XXX  MODIFICATION."  The  difference  between  these  com- 
mands, if  any,  should  be  specified.  Moreover,  it  is  not  apparent  why  the 
system  occasionally  rejected  "MODIFY,”  following  the  system  response: 

"How  do  you  want  to  process  them?"  which  according  to  the  guide  is  ap- 
propriate at  this  point  in  the  search  process. 
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The  guide  should  specifically  state  that  control  must  be  returned 
to  the  system  after  the  end  of  output  for  a query  and  also  after  each 
page  of  printout.  It  is  not  at  all  clear  what  is  required  when  the  sys- 
tem says  "Reply."  There  is  also  a need  to  state,  more  precisely,  that 
if  only  one  sorting  is  desired,  control  must  be  returned.  As  it  is,  the 
initial  reaction  of  all  participants  in  this  evaluation  was  to  think 
there  had  been  an  error  when  the  system  came  back  the  second  time  with 
"ENTER  SORT  SEGMENT,  LENGTH,  MODE." 

Participants  expressed  a general  feeling  that  Appendix  C (review 
questions)  should  have  a greater  proportion  of  practical  operational 
questions,  and  fewer  theoretical,  system,  or  nomenclature  items.  Par- 
ticipants also  indicated  that  the  Sample  Search  Plan  on  page  57  should 
be  expanded  to  include  the  exact  search  request  to  be  input.  Because 
there  is  some  confusion  about  appropriate  procedures  for  using  connectors 
in  compound  requests,  there  would  be  considerable  merit  in  providing  a 
complete  example  in  the  guide.  Indeed,  the  consensus  was  that  a few 
complete  protocols,  at  appropriate  points,  would  improve  the  clarity  of 
instructions  greatly.  This  was  especially  true  for  complex  search 
statements . 

A universal  complaint  was  that  the  recursive  command  "$$WHAT" 

(which  should  provide  a tutorial  message  that  helps  explain  what  the 
user  is  doing  wrong)  generally  failed  to  clarify  the  problem  and,  in 
some  instances,  actually  contributed  to  further  operator  confusion.  It 
was  felt  that  truly  useful  tutorials  should  not  simply  indicate  the  na- 
ture of  the  error  but,  more  important,  should  offer  specific,  detailed 
guidance  on  corrective  actions. 

The  suggested  solutions  for  questions  in  Appendix  E were  a potential 
source  of  confusion  primarily  because  the  abbreviated  format  bore  no  re- 
semblance to  the  correct  method  for  inputting.  Participants  felt  that 
all  references  to  message  entry  format  should  appear  in  the  guide  ex- 
actly as  they  would  be  entered  on  the  keyboard. 


CONCLUSIONS 

There  was  unanimous  agreement  that  the  inclusion  of  an  index  would 
appreciably  reduce  the  amount  of  time  spent  looking  up  information  in 
the  guide.  Users  indicated  frequent  frustration  when  explanatory  infor- 
mation on  a known  topic  was  not  able  to  be  located  rapidly.  This  was 
experienced  particularly  when  participants  were  working  at  a terminal. 

It  was  felt  that  the  guide  would  be  improved  by  having  several  ex- 
ercises in  which  a user  is  stepped  through  a problem,  e.g.,  shown  what 
happens  as  a search  is  expanded  or  narrowed,  or  shown  how  to  truncate 
and  browse  sensibly.  A section  on  common  errors  and  how  to  avoid  them 
would  be  equally  profitable. 
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Among  the  system  features  listed  as  most  desirable  were  (a)  the 
addition  of  a break  command  which  would  permit  users  to  enter  a new  re- 
quest whenever  the  system  was  taking  an  inordinate  amount  of  time  pro- 
cessing a request;  (b)  the  ability  to  use  arithmetic  operators  in  segment 
28;  and  (c)  the  possibility  of  asking  the  system  to  look  for  certain 
words  in  more  than  one  segment  without  having  to  repeat  the  terms  or 
segment  numbers. 


APPENDIX  A 


ADIT  OPERATION  TEST 

This  is  a job-performance  test  intended  for  evaluating  proficiency 
in  the  use  of  ADIT.  It  should  be  completed  within  about  three  hours. 

To  take  the  test  you  will  need  (1)  this  test  form,  (2)  an  ADIT  Search 
Plan  form,  (3)  some  scratch  paper  and  (4)  use  of  a computer  terminal 
with  associated  printer  and  capability  of  reaching  the  ADIT  computer. 

In  completing  the  test,  first  go  through  the  questions,  in  sequence, 
planning  the  necessary  searches  and  displays.  Record  these  in  the 
spaces  provided  following  the  questions  (on  the  Search  Plan  for  Item 
10).  Then  go  to  the  terminal  and  carry  out  the  necessary  activities 
there.  When  printouts  are  requested,  print  only  the  first  page.  Save 
the  complete  printout  of  your  activities  and  displays,  but  record  your 
answers  on  this  form  where  requested.  If  you  search  or  display  differently 
than  you  had  planned,  circle  the  appropriate  symbol  "S"  or  "D"  to  indicate 
that.  Finally,  give  the  test,  your  search  plan,  and  the  complete  printout 
to  the  person  responsible  for  the  test. 

Your  test  will  be  scored  by  a point  system  in  which  items  range 
from  3 to  6 points  in  value  and  the  maximum  obtainable  is  60.  Record 
your  times  below: 

Started  planning:  Started  terminal: 

Stopped  planning:  Stopped  terminal: 


1. 


Get  a chronological  listing  of  items  authored  by  Gagne. 


2.  How  many  records  in  ADIT  from  AC  4000  to  AC  6000  inclusive  were 
published  in  1950? 

S: 


D: 

Answer : 

3.  Get  the  references  to  some  good  quality  general  discussions  of 
transfer  of  training.  List  them  alphabetically  by  author. 

S: 

D: 

4.  What  records  involved  use  of  the  Army  General  Classification  Test 
(ACCT)?  Are  any  of  the  source  documents  in  the  Defense  Documenta- 
tion Center  collection?  If  so,  include  the  DDC  number  with  the 
ADIT  AC  number. 

S: 

D: 

Answer : 

5.  What  descriptive  terms  best  represent  the  topics  of  AC  4170? 

S: 

D: 

Answer : 
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6.  What  additional  research  do  authors  of  articles  on  feedback  after 
1973  suggest  needs  to  be  done?  Make  a single  listing  of  only  those 
references  which  do  provide  suggestions,  and  include  the  suggestions. 
S: 

D: 

7.  What  did  Bourne  and  Archer  conclude  in  1956  from  their  research  on 
distribution  of  practice?  How  did  the  abstractor  feel  about  their 
article? 

A: 

D: 

Answer: 

8.  Was  the  1973  article  by  Levin  on  cue  validity  government-supported? 

If  so,  by  what  organization? 

S: 

Answer: 

9a.  How  many  records  in  ADIT  have  titles  suggesting  that  they  concern 
programmed  learning? 

S: 


9b.  How  many  of  those  are  research  studies? 

S: 

D: 

Answer: 

9c.  Prepare  a bibliography  of  those  programmed  learning  studies  which 
are  concerned  with  technical  training. 

S: 

D: 

9d.  Now  prepare  a bibliography  of  any  programmed  learning  research 
which  involved  comparisons  with  computer  aided  instruction. 

S: 

D: 

10.  Make  up  and  carry  out  a detailed  search  plan  to  obtain  a bibliography 
of  writings  concerned  with  individualized  training.  Submit  the 
plan  with  your  test  and  printout. 
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APPENDIX 


Question  3:  Rate  the  adequacy  of  the  record  Question  4:  Rate  the  adequacy  of  the  record  obtained 


EVALUATION  OF  ADIT 

Question  5:  Assuming  access  readily  available,  estimate  the  frequency 
and  total  time  you  would  use  ADIT  in  one  month. 


SUBJECT 

FREQUENCY 

TIME 

1 

1 

45  minutes 

2 

2-3 

3 hours 

3 

8 

1 1/2  hours 

4 

2 

1 1/2  hours 

5 

2 

4 hours 

6 

1 

1 hour 

7 

1 

5 minutes 

EVALUATION  OF  ADIT 


Question  6:  Think  of  a topic  relating  to  the  design  of  instruction  with 
which  you  have  some  familiarity.  Now  estimate  how  long  it  would  take  a 
person  unfamiliar  with  that  topic  to  do  a literature  review  at  your 
local  libraries,  with  and  without  prior  DDC  search,  and  using  ADIT. 

Subj  1 - Topic:  Training  Devices  and  Simulators 

Library  Search  Time:  30  hours  With  DDC:  8 hours 

ADIT  Search  Time:  2 hours 

Subj  2 - Topic : Compressed  Speech 

Library  Search  Time:  40  hours  With  DDC:  30  hours 

ADIT  Search  Time:  8 hours 

Subj  3 - Topic : Long-Term  Motor  Learning 

Library  Search  Time:  40  hours  With  DDC:  __ 

ADIT  Search  Time:  8 1/2  hours 

Subj  4 - Topic : Multi  Media 

Library  Search  Time:  40-60  hours  With  DDC:  8-10  hours 

ADIT  Search  Time:  2-3  hours 

Subj  5 - Didn't  respond 

Subj  6 - Topic : Laboratory  vs  Lecture  Instruction 

Library  Search  Time:  16  hours  With  DDC:  8 hours 

ADIT  Search  Time:  8 hours 

Subj  7 - Topic : Job  Aids 

Library  Search  Time:  3 hours  With  DDC:  20  minutes 

ADIT  Search  Time:  13  minutes 


EVALUATION  OF  ADIT 


14.12  a = 50.08 


EVALUATION  OF  GUIDE  TO  USE 


EVALUATION  OF  GUIDE  TO  USE 


EVALUATION  OF  GUIDE  TO  USE 


EVALUATION  OF  ADIT  GUIDE  TO  USE 


EVALUATION  OF  ADIT  GUIDE  TO  USE 


Z = 32 
X = 4.6 

a = 2.5 


DISTRIBUTION 


ARI  Distribution  List 


4 OASD  (M&RA)  2 

2 HQDA  (DAMI  CSZ)  1 

1 HQDA  (DAPE-PBR  2 

1 HQDA  (DAMA-AR)  1 

1 HQDA  (DAPE-HRE-PO)  1 

1 HQDA  (SGRD-ID)  1 

1 HQDA  (DAMI  DOT-C)  1 

1 HQDA  (DAPC-PMZ-A)  1 

1 HQDA  (DACH-PPZ-A)  1 

1 HQDA  (DAPE-HRE)  1 

1 HQDA  (DAPE-MPO-C)  1 

1 HQDA  (DAPE-DW)  1 

1 HQDA  (DAPE  HRL)  2 

1 HQDA  (DAPE-CPS)  1 

1 HQDA  (DAFD  MFA)  1 

1 HQDA  (DARD-ARS-P)  1 

1 HQDA  (DAPC-PAS-A)  1 

1 HQDA  (DUSA-OR)  1 

1 HQDA  (DAMO-RQR)  1 

1 HQDA  (DASG)  1 

1 HQDA  (DA10-PI)  1 

1 Chief.  Consult  Div  (DA-OTSG),  Adelphi,  MD  1 

1 Mil  Asst.  Hum  Res.  ODDR&E,  OAD  (E&LS)  1 

1 HQ  USARAL.  APO  Seattle.  ATTN:  ARAGP-R  1 

1 HQ  First  Army.  ATTN:  AFKA-OI  Tl  1 

2 HQ  Fifth  Army,  Ft  Sam  Houston  1 

1 Dir.  Army  Stf  Studies  Ofc.  ATTN:  OAVCSA  (DSP)  1 

1 Ofr  Chief  of  Stf,  Studies  Ofc  1 

1 DCSPER  ATTN:  CPS/OCP  1 

1 The  Army  Lib,  Pentagon,  ATTN:  RSB  Chief  1 

1 The  Army  Lib,  Pentagon,  ATTN:  ANRAL  3 

1 Ofc,  Asst  Sect  of  the  Army  (R&D)  1 

1 Tech  Support  Ofc,  OJCS  1 

1 USASA.  Arlington,  ATTN:  I ARD  T 1 

1 USA  Rsch  Ofc,  Durham,  ATTN:  Life  Sciences  Dir  1 

2 USARI EM,  Natick.  ATFN:  SGRD-UE-CA  1 

1 USATTC.  Ft  Clayton.  ATTN:  STETC  MO-A  1 

1 USAIMA.  Ft  Bragg.  ATTN:  ATSU  CTD-OM  1 

1 USAIMA.  Ft  Bragg.  ATTN:  Marquat  Lib  1 

1 US  WAC  Ctr  & Sch,  Ft  McClellan.  ATTN:  Lib  1 

1 US  WAC  Ctr  & Sch.  Ft  McClellan,  ATTN:  Tng  Dir  1 

1 USA  Quartermaster  Sch,  Ft  Lee,  ATTN:  ATSM-TE  1 

1 Intelligence  Material  Dev  Ofc,  EWL,  Ft  Holabird  1 

1 USA  SE  Signal  Sch,  Ft  Gordon,  ATTN:  ATSO-EA  1 

1 USA  Chaplain  Ctr  & Sch,  Ft  Hamilton.  ATTN:  ATSC  TE-RD  1 

1 USATSCH,  Ft  Eustis,  ATTN:  Educ  Advisor  1 

1 USA  War  College.  Carlisle  Barracks,  ATTN:  Lib  1 

2 WRAIR,  Neuropsychtatry  Div  2 

1 DLI,  SDA,  Monterey  1 

1 USA  Concept  Anal  Agcy,  Bethesda.  ATTN:  MOCA  WGC  1 

1 USA  Concept  Anal  Agcy,  Bethesda,  ATTN:  MOCA-MR  1 

1 USA  Concept  Anal  Agcy.  Bethesda,  ATTN:  MOCA-JF  1 

1 USA  Artie  Test  Ctr,  APO  Seattle.  ATTN:  STEAC-MO  ASL  1 

1 USA  Artie  Test  Ctr.  APO  Seattle,  ATTN:  AMSTE-PL-TS  1 

1 USA  Armament  Cmd,  Redstone  Arsenal,  ATTN:  ATSK-TEM  1 

1 USA  Armament  Cmd.  Rock  Island.  ATTN:  AMSAR-TDC  1 

1 FAA  NAF EC,  Atlantic  City,  ATTN:  Library  1 

1 FAA-NAFEC.  Atlantic  City,  ATTN:  Hum  Engr  Br  1 

1 FAA  Aeronautical  Ctr,  Oklahoma  City,  ATTN:  AAC-44D  2 

2 USA  Fid  Arty  Sch,  Ft  Sill.  ATTN:  Library  2 

1 USA  Armor  Sch.  Ft  Knox,  ATTN:  Library  1 

1 USA  Armor  Sch,  Ft  Knox.  ATTN:  ATSB-DI-E  1 

1 USA  Armor  Sch.  Ft  Knox.  ATTN:  ATSB-DT-TP  1 

1 USA  Armor  Sch.  Ft  Knox.  ATTN:  ATSB  CD  AD  1 


HQUSACDEC,  Ft  Ord,  ATTN:  Library 

HQUSACDEC,  Ft  Ord,  ATTN:  ATEC-EX-E-Hum  Factors 

USAEEC,  Ft  Benjamin  Harrison.  ATTN:  Library 

USAPACDC,  Ft  Benjamin  Harrison,  ATTN:  ATCP— HR 

USA  Comm-Elect  Sch,  Ft  Monmouth,  ATTN:  ATSN— EA 

USAEC,  Ft  Monmouth.  ATTN:  AMSEL-CT-HDP 

USAEC,  Ft  Monmouth,  ATTN:  AMSEL-PA-P 

USAEC,  Ft  Monmouth,  ATTN:  AMSEL-SI-CB 

USAEC,  Ft  Monmouth,  ATTN:  C,  Facl  Dev  Bi 

USA  Materials  Sys  Anal  Agcy,  Aberdeen,  ATTN:  AMXSY-P 

Edgewood  Arsenal,  Aberdeen,  ATTN:  SAREA  BL— H 

USA  Ord  Ctr  & Sch,  Aberdeen,  ATTN:  ATSL  -TEM-C 

USA  Hum  Engr  Lab,  Aberdeen,  ATTN:  Library/Dir 

USA  Combat  Arms  Tng  Bd,  Ft  Benning,  ATTN:  Ad  Supervisor 

USA  Infantry  Hum  Rsch  Unit.  Ft  Benning,  ATTN:  Chief 

USA  Infantry  Bd,  Ft  Benning,  ATTN:  STEBC— TE— T 

USASMA,  Ft  Bliss,  ATTN:  ATSS-LRC 

USA  Air  Def  Sch,  Ft  Bliss.  ATTN:  ATSA-CTD-ME 

USA  Air  Def  Sch,  Ft  Bliss.  ATTN:  Tech  Lib 

USA  Air  Def  Bd,  Ft  Bliss,  ATTN:  FILES 

USA  Air  Def  Bd,  Ft  Bliss.  ATTN:  STEBD-PO 

USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Lib 

USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  ATSW-SE-L 

USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Ed  Advisor 

USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  DepCdr 

USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  CCS 

USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCASA 

USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACO-E 

USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACC-CI 

USAECOM,  Night  Vision  Lab,  Ft  Belvoir,  ATTN:  AMSEL-NV—  SD 

USA  Computer  Sys  Cmd,  Ft  Belvoir,  ATTN:  Tech  Library 

USAMERDC.  Ft  Belvoir,  ATTN:  STSFB-DQ 

USA  Eng  Sch,  Ft  Belvoir,  ATTN:  Library 

USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL  -TD— S 

USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  STINFO  Center 

USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL-GSL 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  CTD-MS 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATS— CTD— MS 

USA  Intelligence  Ctr  & Sch.  Ft  Huachuca,  ATTN:  ATSI— TE 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI— TEX— GS 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI— CTS-OR 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI-CTD-DT 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI—  CTD-CS 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  DAS/SRD 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI-TEM 

USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  Library 

CDR,  HQ  Ft  Huachuca,  ATTN:  Tech  Ref  Div 

CDR,  USA  Electronic  Prvg  Grd.  ATTN:  STEEP-  MT—S 

CDR,  Project  MASSTER,  ATTN:  Tech  Info  Center 

Hq  MASSTER.  USATRADOC,  LNO 

Research  Institute,  HQ  MASSTER,  Ft  Hood 

USA  Recruiting  Cmd,  Ft  Sherdian,  ATTN:  USARCPM—  P 

Senior  Army  Adv.,  USAFAGOD/TAC,  Elgin  AF  Aux  Fid  No.  9 

HQ  USARPAC,  DCSPER,  APO  SF  96558,  ATTN.  GPPE-SE 

Stimson  Lib,  Academy  of  Health  Sciences,  Ft  Sam  Houston 

Marine  Corps  Inst.,  ATTN:  Dean-MCI 

HQUSMC,  Commandant,  ATTN:  Code  MTMT  51 

HQUSMC,  Commandant.  ATTN:  Code  MPI-20 

USCG  Academy,  New  London,  ATTN:  Admission 

USCG  Academy,  New  London,  ATTN:  Library 

USCG  Training  Ctr.  NY,  ATTN:  CO 

USCG  Training  Ctr,  NY.  ATTN:  Educ  Svc  Ofc 

USCG,  Psychol  Res  Br.  DC.  ATTN:  GP  1/62 

HQ  Mid-Range  Br,  MC  Det,  Quantico,  ATTN:  P&S  Div 
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1  Def  & Civil  Inst  of  Enviro  Medicine.  Canada 
1 AIR  CRESS.  Kensington.  ATTN:  Info  Sys  Br 
1 Militaerpsykologisk  Tjeneste,  Cope h age n 
1 Military  Attache,  French  Embassy.  ATTN:  Doc  Sec 
1 Medecin  Chef,  C.E.R.P.A.— Arsenal.  Toulon/Naval  France 
1 Prin  Scientific  Off,  Appl  Hum  Engr  Rsch  Div,  Ministry 
of  Defense.  New  Delhi 

1  Pers  Rsch  Ofc  Library.  AKA,  Israel  Defense  Forces 
1 Ministers  van  Defensie.  DOOP/KL  Afd  Sociaal 
Psycholcgische  Zaken,  The  Hague.  Netherlands 

1 USA  Air  Def  Sch.  Ft  Bliss,  ATTN:  ATS  A TEM 

1 USA  Air  Mobility  Rsch  & Dev  Lab.  Moffett  Fid.  ATTN:  SAVDL-AS 

1 USA  Aviation  Sch.  Res  Tng  Mgt.  Ft  Rucker.  ATTN:  ATST— T-RTM 

1 USA  Aviation  Sch.  CO,  Ft  Rucker.  ATTN:  ATST— D- A 

I HQ,  DARCOM,  Alexandria.  ATTN:  AMXCD-TL 

1 HQ.  DARCOM.  Alexandria.  ATTN:  CDR 

1 US  Military  Academy.  West  Point.  ATTN:  Serials  Unit 

1 US  Military  Academy,  West  Point,  ATTN:  Ofc  of  Milt  Ldrshp 

1 US  Military  Academy,  West  Point.  ATTN:  MAOR 

1 USA  Standardization  Gp.  UK.  FPO  NY.  ATTN:  MASE-GC 

1 Ofc  of  Naval  Rsch.  Arlington.  ATTN:  Code  452 

3 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  458 

1 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  450 

1 Ofc  of  Naval  Rsch.  Arlington.  ATTN:  Code  441 

1 Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN:  Acous  Sch  Div 

1 Naval  Aerospc  Med  Res  Lab,  Pensacola.  ATTN:  Code  L51 

1 Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN:  Code  L5 

1 Chief  of  NavPers,  ATTN:  Pers-OR 

1 NAVAIRSTA.  Norfolk.  ATTN:  Safety  Ctr 

1 Nav  Oceanographic.  DC.  ATTN  Code  6251.  Charts  & Tech 

1 Center  of  Naval  Anal.  ATTN:  Doc  Ctr 

1 NavAirSysCom.  ATTN:  AIR-5313C 

1 Nav  BuMed.  ATTN:  713 

1 NavHelicopterSubSqua  2.  FPO  SF  96601 

1 AFHRL  (FT)  William  AFB 

1 AFHRL  (TT)  Lowry  AFB 

1 AFHRL  (AS)  WPAFB.  OH 

2 AFHRL  (DOJZ)  Brooks  AFB 

1 AFHRL  (DOJN)  Lackland  AFB 
1 HOUSAF  (INYSD) 

1 HOUSAF (DPXXA) 

1 AFVTG  (RD)  Randolph  AFB 

3 AMRL  (HE)  WPAFB.  OH 

2 AF  Inst  of  Tech.  WPAFB.  OH.  ATTN:  ENE/SL 
1 ATC  (XPTD)  Randolph  AFB 

1 USAF  AeroMcd  Lib.  Brooks  AFB  ISUL-4)  ATTN:  DOC  SEC 
1 AFOSR  (NL).  Arlington 

1 AF  Log  Cmd.  McClellan  AFB.  ATTN:  ALC/DPCRB 

1 Air  Force  Academy.  CO.  ATTN:  Dept  of  Bel  Sen 
5 NavPers  & Dev  Ctr,  San  Diego 

2 Navy  Med  Neuropsychiatric  Rsch  Unit.  San  Diego 
1 Nav  Electronic  Lab.  San  Diego.  ATTN:  Res  Lab 

1 Nav  TrngCen.  San  Diego.  ATTN:  Code  9000-  Lib 
1 NavPostGraSch,  Monterey.  ATTN:  Code  55Aa 
1 NavPostGraSch,  Monterey.  ATTN:  Code  2124 
1 NavTrngEqjipCtr.  Orlando.  ATTN:  Tech  Lib 
1 US  Dept  of  Labor,  DC,  ATTN-  Manpower  Admin 
1 US  Dept  of  Justice.  DC,  ATTN.  Drug  Enforce  Admin 
1 Nat  Bur  of  Standards.  DC.  ATTN:  Computer  Info  Section 
1 Nat  Clearing  House  for  MH  — Info,  Rockville 
1 Denver  Federal  Ctr.  Lakewood,  ATTN  BLM 
1?  Defense  Documentation  Center 

4 Dir  Psych.  Army  Hq,  Russell  Ofcs,  Canberra 

1 Scientific  Advsr,  Mil  Bd,  Army  Hq,  Russell  Ofcs,  Canberra 
1 Mil  and  Air  Attache.  Austrian  Embassy 

1 Centre  dr?  Recherche  Des  Far  tours.  Humame  de  la  Defense 
Nationale.  Brussels 

2 Canadian  Joint  Staff  Washington 

1 C/Au  Staff.  Royal  Canadian  AF.  ATTN:  Pers  Std  Anal  Br 

3 Chief.  Canadian  Def  Rsch  Staff.  ATTN:  C/CRDS(W) 

4 British  Def  Staff.  British  Embassy.  Washington 


1 US  Marine  Corps  Liaision  Ofc,  AMC.  Alexandria.  ATTN:  AMCGS— F 
1 USATRADOC.  Ft  Monroe  ATTN:  ATRO-ED 
6 USATRADOC.  Ft  Monrc  ATTN:  ATPR-AD 
1 USATRADOC.  Ft  Monroe.  ATTN:  ATTS-EA 

1 USA  Forces  Cmd.  Ft  McPherson.  ATTN:  Library 

2 USA  Aviation  Test  Bd.  Ft  Rucker.  ATTN:  STEBG-PO 

1 USA  Agcy  for  Aviation  Safety.  Ft  Rucker,  ATTN:  Library 
1 USA  Agcy  for  Aviation  Safety,  Ft  Rucker.  ATTN:  Educ  Advisor 
1 USA  Aviation  Sch,  Ft  Rucker,  ATTN:  PO  Drawer  O 

1 HQUSA  Aviation  Sys  Cmd.  St  Louis,  ATTN:  AMSAV-ZDR 

2 USA  Aviation  Sys  T«t  Act..  Edwards  AFB.  ATTN:  SAVTE-T 
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